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ABSTRACT 

A comprehensive conceptual framework for the 
validation of educational programs has been proposed. The conceptual 
framework includes the concept of an educational program as a system 
of curricular and instructional claims and methods of evaluation 
which entail a wide range of judgments of both logical validity and 
factual validity. Judgments of logical validity are made in reference 
to the internal structure of each of the claims and its various 
components, the relationships among the claims constituting a 
program, and the relationships between the component claims of a 
program and the bases of the profession under which the program is 
formulated and operates. The logic of reaching judgment^ of factual 
validity of the individual curricular and instructional claims, and 
of the program as a whole, has been described. Judgments of factual 
validity of the various component claims, and of the program, are 
made on the basis of observations of pupil performance and teacher 
action to determine whether those behaviors correspond to that which 
is entailed by fulfillment of the goal, rule, and qualifier 
components of the claims. (Author) 



* Documents acquired by ERIC include many informal unpublished * 

* materials not available from other sources. ERIC makes every effort * 

* to obtain the best copy available. Nevertheless, items of marginal * 

* reproducibility are often encountered and this affects the quality * 

* of the microfiche and hardcopy reproductions ERIC makes available * 

* via the ERIC Document Reproduction Service (EDRS). EDRS is not * 

* responsible for the quality of the original document. Reproductions * 

* supplied by EDP.S are the best that can be made from the original. * 



EKLC 



co 
rvj 



THE VALIDATION OF EDUCATIONAL PROGRAMS 



Emajean McCray 
Department of Research and Evaluation 
Fort Worth Independent School District 

arid 

John Lottes 
School of Education 
Texas Christian University 

March, 1976 



00 



in 




us OEPARTMENTOF HEALTH. 
EDUCATION A WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS OOCUVENT HAS BEEN REPRO- 
OUCEO EXACTLY AS RECEJVEO FROM 
fHE PERSON OR ORGANIZATION ORIGIN- 
ATING »T POINTS OF VIE* OR OPINIONS 
STATEO DO NOT NECESSARILY REPRE- 
SENT OFFICIAL NATIONAL INSTITUTE OF 
EOUCATlON POSITION OR POLICY 



This paper has been developed for presentation at 
the April 1976 American Educational Research Associ- 
ation conference in San Francisco, California 



ERIC 



PRESENT STATE OF EDUCATlO^^^ ^fioG^^^^ ^^^LVA'f'^^^ ' ♦ . . ^^^^ ^ 
PURPOSE AND ORGANIZATION ^^XS ^ 

THE CONCEPT OF AN EDUCA^I^^^^ f>RO(^^^^ 



Toward an Adequate CoJ'^^^^ , p^tiofi^ °9i>aff? 
The Stvuatuve of a-^ ti^^'^tio^U t'>,°9:faif I 
The Functions of ctn ^ ^t:iof^^ °9'i'aff' 

CURRICULAR CLAIMS • ' ^ .,,.''*'•..,,'''*''•..,''' ' 

INSTRUCTIONAL CLAIMS ^ ' ^® 

THE CONCEPT OF VALIDATIC^ ^ ' '''«.,,''''''•..,.'''' 

The Meaning of Val-ida'' h 

Validation ^l^^Xt. 
Logical Validity and ^x^^i 

Significance of th& ^X>± OJ '^'iati^ 



PRINCIPLES OF 
OVERVIEW 

A PRIORI ASPECTS OF EDVC^^'^ "''t ^0^'"^'^ Vj^P' 



2( 



So 

Judgments of Meaniri9 ' 3^ 

Judgments of System^^ ^ ^^^^rt ^ Vais 3» 

Judgments of Poten-ti^^ ^^tu^^ ^^^ty 

A POSTERIORI ASPECTS OP PP^^^^:CoHA^ ^^^^^AM ^^^'^^^^Tofl 

The Logic of Testing ^ ^^^i^^ '^aim 

Decision Rules for ^^^^^oV , I'aZt^ifaiw ^® 

of Singular Curricu^^f ^ ^Onat 



Decision Rules for Judging Factual Validity 41 
Of an ^d'^^ational Program 

SUMMARY AND ASSESSMENTS 

SUMMARY SO 

ASSESSMENTS OF TRE PROPOSED CONCEPTS AND PRINCIPLES 51 

A CONCLUDlf^G NOTE 53 



4 



INTRODUCTION 
Present State Of Educational Program Evaluation 



Each year there are volumes of evaluation reports written 
on the many diverse programs being offered in our educational 
institutions. This abundance of reports demonstrates that 
sizable investments are being made in time, effort, and money 
in order to evaluate current programs- However, the prolific 
activity alone has not produced a sufficiently comprehensive 
range of evaluative judgments nor warranted inferences as to 
program effectiveness. As a result, evaluation reports are 
often regarded as being not very useful in helping to make de- 
cisions about educational programs. 

Daniel Stufflebeam, in an address delivered at the Working 
Conference on Assessment Theory, noted that "The most basic . - - 
[problem] is a lack of adequate theory or conceptualizations 
pertaining to the nature of evaluation which [is] needed to 
accommodate educational programs" ([20], p- 8). 

The authors of this paper believe that one fundamental 
reason why program evaluation is inadequate is that there is a 
lack of continuity between education and the well-developed 
disciplines (especially, philosophy of science, value theory, 
logic, theory of action and philosophy of language). Educational 
evaluators have failed to utilir.e, to a sufficient degree, the 



5 



2 



logical and empirical principles ir which evaluations of scien- 
tific systems are grounded. Moreover, they have — by creating 
a gulf between the language of educational evaluation and the 
language of evaluation utilized in the natural and behavioral 
sciences, including their philosophical foundations inadver- 
tently cut off the routes to adequate educational evaluation. 
The resultant crippling limitations in educational program eval- 
uation include (a) inadequate concepts of 'educational program', 
'curriculum' and 'instruction', (b) an inadequate concept of 
'evaluation' and (c) failure to include judgments of logical 
structure in program evaluation. 

The inadequate concepts of 'educational program', 'curric- 
ulum' and 'instruction' result in a vagueness as to the object 
to be evaluated. Consequently, judgments as to program effec- 
tiveness are — if carefully analyzed — judgments of hypotheses 
whose antecedent conditions are unknown. 

The inadequate concept of 'evaluation' leads to neglect of 
the necessary relationship of value judgments to a credible 
conceptual and value framework, and consequently, to unwarranted 
judgments about educational programs. 

The absence of judgments of logical structure in program 
evaluation results in such defects as (a) failure to take into 
account auxiliary hypotheses that are implicitly asstxmed to be 
valid, and (b) judgment of the validity of hypotheses on the 
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basis of non-relevant observations. 

Purpose And Orcjanization Of This Patter 

The purposes of this paper are to (1) construct an adequate 
concept of 'educational program', (2) formulate principles for 
the validation of educational programs in such fashion as to 
include both logical and empirical considerations, and (3) ex- 
amine the practical implications of these concepts and prin- 
ciples in reference to educational program development, accep- 
tance, and improvement • 

First, the definition of an 'educational program' will be 
set forth in terms of the concepts 'curricular claim' and 
'instructional claim.' The extra-logical components of a cur- 
ricular claim will be described and illustrated; and the rela- 
tionship between 'curricular claim' and 'instructional claim' 
will be clarified. 

Second, the concept of 'validation' will be set forth in 
conjunction with the related concepts of 'logical validity' and 
'factual validity.' Validity, in both its logical sense and 
its empirical sense, will be construed as a judgment about 
sentences based on relations between sentences* 

Third, principles for the validation of educational pro- 
grams will be set forth; both logical and empirical principles 

7 



4 



will be included. These principles will take into account 
a priori judgments of validity and a posteriori judgments of 
validity. 

Fourth, the pragmatics of educational program validation 
will be examined in reference (a) to the concepts set forth 
in this paper, and (b) to educational program development, 
acceptance , and improvement . 
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CONCEPTUAL ORIENTATIONS 

The Concept Of An Educational Program * 

Toward An Adequate Concept Of Educational Program 

The way in which a person conceptualizes an educational 
program is strongly influenced by his value orientations. The 
value orientations expressed by statements of this paper are 
rooted in the premise that education is (or ought to be) gov- 
erned by the values of rationality, objectivity, and indepen- 
dent judgment. This premise is supported by such documents as 
the NEA Code of Ethics [11] and Isr?el Scheffler's ([18], p. 11) 
treatment of rationality as the demarcation criterion for dis- 
tinguishing teaching from other influence activities such as 
"deception, insinuation, advertising, propaganda, indoctrination, 
suggestion, bribery, and force." 

Under this perspective, the concept of an 'educational 
program' should be formulated in such a way as to: 

(a) Take into account (1) ends to be achieved, 
(2) the means by which the ends are achieved, 
and (3) the circumstances under which the 
means are reasonably applicable; and 

(b) Render the beliefs as to the effectiveness 
of the means, in reference to achievement of 
the ends, amenable to public verification. 
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To fulfill these requirements, the beliefs to be verified 
must be articulated in explicit fashion, i.e. in statement 
form. At least some statements must be set forth as relation- 
ships between the educational ends to be achieved, the means by 
which they are achieved, and the conditions under which the 
means may be reasonably invoked; that is, some statements must 
be articulated in the form of empirically testable propositions, 

Moreover, it is crucial to avoid confounding the beliefs 
under evaluation with the observational bases in which judg- 
ments as to the factual validity of those beliefs are grounded. 

The Structure Of An Educational Program 

An educational program will be construed as a system of 
statements. The system is composed of propositions, at least 
some of which are curricular claims, and some of which may be 
instructional claims.* Each claim links three components: 
(a) a goal component (ends), (b) a rule component (means), and 
(c) a qualifier component (conditions under which the means 
are believed effective for achieving the ends). That is, each 
claim specifies a relation between some course of action, de- 
fined by a set of rules, and attainment of an intended goal- 
state by an individual (pupil); and includes specification of 
the set of individuals for which the relation is believed to 

* It may be helpful for the reader to familiarize himself with the 
sections on the concepts of "curricular claims" (see page 11) 
and "instructional claims" (see page 1^) • 
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be valid. 

Currlcular claims, or curricular claims in conjunction 
with instructional claims, constitute the parts of any system 
of statements which fulfills the concept of 'educational pro- 
gram. ' Adequate description of a system requires not only 
identification of its parts, but also identification of various 
connections among those parts. Consequently, we must delineate 
the interrelationships within any system of curricular and 
instructional claims which warrants the denotation of '^educa- 
tional program." 

The set of curricular claims may consist entirely of claims 
that function as primitive sentences and claims which 'are 
derived from the claims functioning as primitives. Or the set 
may consist of curricular claims which function as primitive 
sentences, while other claims are partially derived from the 
primitive curricular claims; in this case, auxiliary hypotheses 
account for the additional information utilized in the derived 
curricular claims . 

Under this perspective, representation of an educational 
program requires descriptive statements of at least two dif- 
ferent levels of specificity. At one level the program ends, 
program organizational rules, and program qualifying conditions 
must be articulated within the context of higher order curric- 
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ular claims (i.e. "program-level** claims). At a more specific 
level the ends, procedural rules and qualifying conditions of 
each curricular component (e.g. "course" component) must be 
articulated within the context of a more specific curricular 
claim; these more specific curricular claims must, of course, 
be at least partially derived from the "program-level" claim 
or claims. 

In some cases, representation of an educational program 
should be extended to include articulation of certain instruc- 
tional claims. Only instruct iqnal claims whose range of appli- 
cation is synonomous with that of the relevant curricular claim 
are included within the concept of "educational program." In- 
clusion of only universally applied instructional claims within 
the system of claims constituting a program, makes it sensi- 
ble to construe an educational program in a singular sense 
rather than as a variable whose replacement values cannot be 
ascertained . 

The set of instructional claims bears a compatibility 
relation in reference to the set of curricular claims; i.e. 
no instructional claim may be logically inconsistent nor em- 
pirically non-relevant to the set of curricular claims. The 
substantive elements with which each instructional claim is 
designed to deal, must be contained within the substantive 
units specified by the curricular rules. 
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kinds must occur before initiation of empirical tests. 

From a third perspective, t*i function of an educational 
program (defined as a system of claims), is to make interpret- 
able empirical tests possible. One way in which this is ac- 
complished is by representing an educational program as a set 
of related statements. This makes it possible to satisfy the 
condition of reproducibility, i.e. to achieve reproducibility 
of treatments and outcomes over different occasions and 
evaluators . 

Representation of an educational program as a set of 
statements also makes it possible to separate that which is 
being tested from the observational bases for the evaluative 
judgments made. (For further explication, see the subsequent 
sections on curricular and instructional claims and prin- 
ciples of educational program validation.) 
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Currioul'ar Claims * 

Each curricular claim is a proposition which links three 
components: (a) a goal component, (b) a rule component, and 
(c) a qualifier component. A curricular claim specifies a 
relation between some course of action, defined by a set of 
rules, and attainment of an intended goal-state by an indi- 
vidual (pupil); and includes specification of the set of indi- 
viduals for which the relation is believed to be valid. A 
curricular claim will be viewed as an hypothesis to which both 
teacher(s) and pupils subscribe, and under which both act. 

A generalized curricular claim might be represented in 
the following way: 

Generalized Curricular Claim 

For each pupil Xy where X satisfies conditions C; 
If both the teacher (s ) and a set of pupils, 
of which X is a member, act under rules 
then X will (probably ) attain goal^state S. 

The 'generalized curricular claim', as used here, is a 
proposition generator, i.e. a curricular claim generator, but 
itself cannot be judged true or false. The 'curricular claim,' 
however, denotes a proposition which can be judged true or false. 

A curricular claim contains both logical terms, e.g. 'if,' 

•then,' and extra-logical components which are linked by the 

logical connectors. The extra-logical components will be 

* This section on curricular claims is drawn from the paper, ''The Nature and 
Significance of Curricular Claims and How They Are Validated'' (John Lottes 
. and Emajean McCray) , presented at the 1975 AERA Conference 4 [131* 

15 



labeled: ourricular goal-state^ ourrioular rules^ and cwrric- 
ular qualifying conditions . These linguistic structures pro- 
vide the organizational framework for teaching and learning 
activity. Simple examples of these extra-logical components 
are displayed within the context of the illustrative curric- 
ular claim in Figure 1. 

The ourrioular goal-state is represented in the form of 
a set of concepts or sentences (inclusive sense of 'or'), and 
denotes some desired individual pupil-state. This representa- 
tion must have empirical import: the state of affairs denoted 
by the 'curricular state' must, in principle, be capable 

of realization and of meaningful test to determine whether its 
conditions have been fulfilled. 

Currioular rules are rules which govern the actions [2] 
of both teacher and pupil. They 'rule in' some acts and 'rule 
out' other acts. Still other acts (such as chewing gum) may 
be value-free in reference to the rules. Ordinarily, curricu- 
lar rules (ought to) include a substantive aspect and also a 
regulative aspect . 

We will construe the substantive aspect of a set of cur- 
ricular rules in such a way^s to include: 

(a) Identification of the basic elernents^ units, 
or parts with which the participants (i.e. 
teacher and pupil) will deal. These basic 
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units may be concepts or principles, problems, 
activities, aesthetic objects, etc.; and 

(b) I^^^tzfication of the interrelationships among 
the basic elements y uni tSj or parts with which 
the participants will deal. 

The regul^-^i^e aspect of a set of curricular rules spec- 
ifies the ways in which the teacher and pupil, or pupils, 
should act in relation to each other, and in relation to* the 
basic units of '^^^ enterprise. We will construe the regula- 
tive aspect of curricular rules in a way which includes: 

(a) I^^'^tification of the ways in which actions^ 

lation to the basic curricular units j 
will be ini tiatedj conducted^ terminated^ 
and Validated; and 

(b) I^^^tifioation of that which is obligatory j 
permissible J or not permissible on the part 
of both teacher and pupil. 

The Qux^ri^^^^^ qualifying conditions consist of a set of 
propositions which describe the pupil-state assumed to be 
necessary to effective functioning under the curricular rules, 
and in reference to the curricular goal-state. Of course, the 
pupil must have had prior opportunity of achieving the state 
of affairs denoted by the qualifying conditions since judgment 
as to the factual validity of the propositions, in reference to 
each potential P^PH, must be made before the curricular rules 
are invoked. 

The extra-logical components of a curricular claim govern 
instructional learning activities in much the same way that 
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the rules' of a game govern the activities of all players through 
the course of the game. 

The rules of any game make some actions mandatory for 
each player, some actions permissible, some actions forbidden, 
and other actions value-free; on the other hand, each player 
is allowed some degree of freedom to determine the way in which 
even most mandatory acts will be performed. Consequently, 
each player develops an individual strategy that guides the 
way he perform^ ^.ndividual acts during the course of the game. 
Moreover, each player characte^-istically changes his strategies 
in response to the acts of opposing players. These strategies 
must conform to the rules of the game and, of course, have the 
purpose of achieving the goal of the game. 

In an analogous way, curricular rules make some actions 
obligatory on the part of the pupil or teacher, while making 
some actions permissible, some actions not permissible, and 
other actions value-free in reference to the rules. The cur- 
ricular rules leave both teacher and pupil some degrees of free- 
dom to determine the way in which their individual acts will 
be performed. The teacher develops strategies for influencing 
the pupil toward goal-state attainment, and adapts these strat- 
egies in response to the patterns of individual pupil acts. 
The pupil, deliberately or otherwise, forms strategies which 
guide his individual acts toward goal-attainment, and adapts 
these strategies in response to the patterns of individual 
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teacher acts. 

It is important to recall that v/e are viewing education 
as a cooperative enterprise where both teacher(s) and pupils 
have subscribed to the same curricular rules and goal-state. 
Both teacher strategies and pupil strategies, as well as their 
individual acts, should conform to the common rules - 

Of course a given strategy is reasonable only if its user 
believes that, under the existing circumstances, implementation 
of that strategy will lead to achievement of certain ends. 
Moreover, within the professional context, a teacher's strategy 
is justified only when it is imbedded within the rule component 
of some claim that the teacher has reason to believe is valid. 
This sort of a claim will be called an instructional claim. 

Instructional Claims * 

The concept of ins tructiona I claim is instrumental to 
realization of the various conditions imbedded within a curric- 
ular claim, thus rendering it crucial to curricular claim 
validation. Both teacher and pupil(s) subscribe to, and act 
under, a curricular claim. But, only the teacher subscribes 
to, and acts under, an instructional claim. 

A generalized instructional claim might be represented in 
the following way: 

* The section on instructional claims is drawn from the 1975 AERA paper, 
"The Nature and Significance of Curricular Claims and How They Are 
Validate;'* (John Lottes and Bmajean McCray), [13]. 
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Generalized Instruational Claim 

For each pupi I Xj where X satisfies conditions Ij 
If the teacher (s) acts under rules in relation 
to a set of pupils of which X is a member^ 
then X will (probably ) attain goal^state G. 

The extra-logical components of an instructional claim 
will be called: instructional goal^-state^ instructional rules^ 
and instructional qualifying conditions . Each of these com- 
ponents will be illustrated within the context of the sample 
instructional claim displayed in Figure 2. 

The instructional goal-stqte can be articulated in the 
form of a set of concepts or sentences (inclusive sense of 
*or') which represent some intended individual state of affairs 
to be achieved by the pupil. The instructional goal-state can 
be synonomous to the curricular goal-state, logically or ethi- 
cally necessary to attainment of the curricular goal-state , or 
empirically useful to attainment of the curricular goal-state. 

Instructional rules are represented as a set of rule- 
statements which determine each individual act that the teacher 
will perform in every acting-situation expected to occur. In- 
structional rules constitute a plan, or a set of procedures, 
formulated for the purpose of guiding teacher actions. The 
teacher is free to modify, or replace entirely, any set of in- 
structional rules under which he intends to act. However, 
neither the instructional rules nor consequent teacher acts 
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Figure 2 

AN ILLUSTRATIVE INSTRUCTIONAL CLAIM * 



For each pupil X, where X satiafies these inotructional qualifying conditions: 

1(1): X Comprehends the basic problem-solving paradigm, 

I [2]; X is deliberately aware of, and can utilize, credible sources 
of both causal and teleological explanations of human behavior, 

I[3J; X has ndequatcly resolved at least one problem of Subset A; 



I£the teacher(a) acts under these rules in reference to a pupil aubset of which 
X ia a member: 



T[l): The teacher (T) will propose that the pupil subset, including X, 

assume the responsibility of identifying, and proposing for resolu- 
tion, a problem of practical concern to each pupil of the subrot; 
and, where resolution of a conflict between an individual and a 
group of organized individuals (i.e. organization) is entailed. 

T[2): _I£ the responsibility is accepted by each pupil of the subset, then 
T will follow these rules: 

(a) T will initiate no moves relative to the problem Identi- 
fication, or subsequent resolution, efforts of the pupil 
subset; 

(b) On any occasion that any pupil, or pupils, of the subset 
claiir.s that (1) some step oi* steps toward identification 
or resolution of a significant problem have been adequately 
performed, or (2) the problem has been adequately identified 
or resolved, then T will demand justification of the claim; 
and, if T perceives that the justification in not iidequate, 
then T will illuminate the defect by the Socratic method; 

(c) If any pupil, or pupils, of the subset requests guidance as 
to what moves to make or how to make them, then T will eith- 

(1) suggest potentially useful information sources, or 

(2) provide explanation or illustraticn in a manner that 
minimizes synthesis by T and roaximi2e8 synthesis by the pupil 
or pupils. 

T(3]: If the pupil subset does not accept the proposed responsibility, then 
T will follow this rule: 

First, T will act in a manner governed by the curricular rules 
in attempting to resolve the issue; and, if that fails, then T 
will enter the diagnosis, explanation, remediation mode for 
identifying and reaolving unanticipated classroom problems (un* 
der the curricular rules).; 



then X will (probably) attain this inatructional goal'-atate: 

• The ability to apply credible problem-solving principles 
in some situations involving conflict between an individ* 
oal aod an organized group of individuals. 

* This instructional claim has been constructed in reference 
to the illustrative curricular claim displayed in Figure 1 
and is compatible with that curricular claim. 
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may violate the curricular rules to which both teacher and 
pupil have subscribed. Otherwise, realization of rationality 
would be thwarted; and, empirical test of curricular claims 
could not be achieved. 

The instructional qualifying conditions consist of a set 
of propositions describing the existing state of a pupil. The 
pupil-state is described in reference to (a) the curricular 
rules or curricular goal-state, or (b) some instructional goal- 
state contained within an instructional claim which is adequate 
in reference to the curricular claim (or claims) , or (c) indi- 
vidual pupil goals, strategies, or assumptions under which the 
pupil acts. Instructional qualifying conditions are intra- 
curricular whereas curricular qualifying conditions are extra-- 
curricular. 
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The Concept Of Validation 

The Meanings Of Measurement And Evaluation 

S. S. Stevens, in ''On The Theory of Scales of Measurement" 

[19] , set forth a widely used definition of measurement : 

Measurement is the assignment of numerals 
(i.e. symbols) to objects or events 
according to rules . 

Stevens also distinguished among four different levels of 
measurement : nominal, ordinal , interval, and ratio. Nominal 
and ordinal levels of measurement satisfy certain classif icatory 
or comparative, but non-arithmetic, properties. Interval and 
ratio levels of measurement satisfy the classif icatory and com- 
parative properties, and certain arithmetic properties as well. 

It is fruitful to construe evaluation as a particular kind 
of measurement, and the definition which follows is governed 
by this perspective. 

It is significant to observe that the concept of value is 
imbedded within the term "e valu ation;" i.e. e-valuCe) -ation 
([16] , p. 61). Evaluation will be defined here in such a way 
as to appeal to values in the process of assigning symbols to 
objects of evaluation. 'Value terms' will constitute the 
symbols to be assigned; e.g. 'good/ 'fair,' 'poor,' or 
'adequate,' 'inadequate,' etc. These terms are assigned 
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according to the degree to which certain values are realized 
by the object being judged. 

The following definition of evaluation borrows substantially 
from Nicholas Rescher's Introduction To Value Theory ([16], 
pp. 61-72): 

Evaluation is measurement ^ conducted within a 
value framework^ and resulting in the assign- 
ment of a value term (e.g. good-bad^ adequate- 
inadequate ^ valid-contravalid^ etc. ) to the 
object or event under evaluation. The assign- 
ment (i.e. judgment ) is made on the basis of: 

(1) the extent to which the value (or values) 
is realized in the object or event; or 

(2) the extent to which the object or event 
facilitates realization of the value by 
some other object or event. 

This definition has several important virtues. First, 
its meaning is precise, assuming one uses Rescher's meaning of 
value ( [16J , pp. 8-10) and Stevens* meaning of measurement. 
Second, the relationship between measurement and evaluation 
is defined. Evaluation is a particular kind of measurement; 
it is measurement at ordinal, interval, or ratio level. Third, 
this meaning of evaluation is consistent with the usage of the 
term in mathematics, the sciences, analytic philosophy, and 
other more rigorous disciplines. Fourth, it will be useful 
in our present context. 
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The Meaning Of Validation 

Validation 

i Validation is the process of making judgments of validity. 
Judgments of validity are judgments of statements, and these 
judgments are based on logical (deductive or inductive) rela- 
tions between statements. In validating an educational program, 
we are, of course, interested in making judgments of statements 
of the types that we have labeled 'curricular claims' and 
'instructional claims,' These .judgments of statements may be 
judgments as to logical validity or judgments as to factual 
validity [ 4 ] . 

Logical Val idity And Factual Validity 

The principles of logic constitute grounds for determining 
whether one sentence (i.e. "conclusion") is a consequence of 
another sentence or set of sentences (i.e. premises). If the 
relevant principles are realized, i.e. if the conclusion is a 
justified deductive inference in reference to prior sentences, 
then the concluding sentence is judged to be logically valid 
with respect to those premises. If, under the logical princi- 
ples being applied, the conclusion is demonstrably false, then 
the concluding sentence is judged to be logically contravalid. 
In other cases it may not be possible to make a justified 
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judgment bn the basis of logical rules alone; in this event 
the concluding sentence is logically indeterminate . 

Some sentences are called factual propositions ([7], pp. 
16-21) • Factual propositions are statements that have some 
bearing on empirical objects or events. Factual propositions, 
some of which are called 'hypotheses,' must be judged on the 
basis of their relation with sentences which describe observa- 
tional data. If a factual proposition corresponds with the ob- 
servation sentences, or observation reports, then that set of 
observation sentences is viewed as having confirmed the factual 
proposition to some degree. In this event, and in a very tenta- 
tive sense, the proposition is judged to be factually valid. 
If the set of observation sentences does not correspond with 
the factual proposition, then that proposition is disconfirmed 
to some degree; the factual proposition is then judged tenta- 
tively to be factually contravalid. In reference to observa- 
tion sentences which do not fulfill certain semantical condi- 
tions, it may be impossible to make a justified judgment about 
the factual proposition; in this case the proposition may be 
viewed as factually indeterminate in reference to those obser- 
vation sentences. 

Carl Hempel, in Aspects Of Scientific Explanation [ 5] has 
noted that the relation between an hypothesis and relevant ob- 
servation sentences is a basically semantical relation. Hempel 
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has illustrated this relation within the following explication 
of the view of confirmation as a relation between sentences: 



•'It is possible ... to construe confirmation ... 
as a rel at ion between two sentences , one describ- 
ing the given evidence, the other expressing the 
hypothesis. Thus, instead of saying that an 
object a which is both a raven and black (or the 
fact of a being both a raven and black) confirms 
the hypothesis that all ravens are black, we may 
say that the evidence-sentence, 'a is a raven, 
and a is black* confirms the hypothesis-sentence 
... , 'All ravens are black.' We shall adopt this 
conception of confirmation as a relation bptween 
sentences for the following reasons: ' ^ c, the 
evidence adduced in support or criticism of a 
scientific hypothesis is always expressed in sen- 
tences, which frequently have the character of 
observation reports; .and second, it will prove 
very fruitful to pursue the parallel ... between 
the concepts of oonf ivmation and logical oonse^ 
quenoe. And just as in the theory of the con- 
sequence relation, i.e. in deductive logic, the 
premises of which a given conclusion is a con-^^e- 
quence are construed as sentences rather than as 
'facts,' so we propose to construe the data which 
confirm a given hypothesis as given in the form 
of sentences ([5], pp. 21-22)." 



Under the preceding perspectives, validation is construed 
as the process of making judgments as to the logical validity 
or factual validity (i.e. confirmation) of sentences. More- 
over, judgments of both kinds are made on the basis of rela- 
tions between sentences. Validation is construed here as a 
particular kind of evaluation, just as evaluation is a 
particular kind of measurement. 
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Significance Of The Concept Of Validation 

There are several reasons for introducing the concept of 
validation. First, the concept of validation seems compatible 
with the view of an educational program as a system of 
claims. Validation, as a particular kind of evaluation, is 
concerned with the process of testing such claim systems. 

The second reason is a psychological one. There is a 
need to remove the psychological barriers created when one 
uses the expression "evaluation of educational programs.** 

A third reason for introducing the concept of validation 
is to focus attention on the worth of modelling more exten- 
sively on the language of science; thus creating continuity 
between education and the disciplines. 
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PRINCIPLES OF EDUCATIONAL PROGRAM VALIDATION"^ 

Overview 

An educational program has been conceptualized as a 
system of claims. This system includes at least some curric- 
ular claims, and may include instructional claims as well. 
Meanings of the extrarlogical components of the curricular 
and instructional claims have been set forth and illustrated. 
The functions of this linguistic structure are to guide teach- 
ing and learning activity, and consequently, to provide the 
bases for making valid judgments of validity. 

Educational program validation, in the sense used here, 
entails making a wide range of judgments of validity. These 
include judgments made before the educational program is act- 
ually submitted to empirical test. Such judgments are made in 
order to determine whether the educational program warrants 
empirical trial in the first place. Assuming that the program 
passes these various tests of its potential worth, it is then 
reasonable to submit it to empirical trials of its various 
predictive implications . 

Explication of these two crucial aspects of educational 
program validation is provided in the succeeding sections. 
First, we will consider a priori aspects of educational program 

* The principles of e4ucational program validation set forth in this section 
are based upon prior papers by John Lottes [9]/ and John Lottes and Ema-"* 
jean McCray [131; the present paper explicates the basic principles in a 
more precise and comprehensive way, and formulates new principles that 
apply only to systems of curricular and instructional claims. 
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validation; second, we will consider a posteriori aspects of 
educational program validation. A priori refers to those 
aspects of validation conducted prior to the decision as to 
whether predictive tests of factual validity are warranted. 
A posteriori refers to those aspects of validation conducted 
consequent to such a decision (assuming that the a priori 
judgments are favorable to the educational program). 

A Priori Aspects of Educational Program Validation 

Three different kinds of judgments should be made prior 
to the decision as to whether a proposed educational program 
merits empirical trial. First, judgments should be made as to 
whether the educational program has adequate meaning. Second, 
judgments should be made as to the systemic import* of the educa 
ticnal program. Third, judgments should be made as to the po- 
tential factual validity of the educational program. All of 
these judgments are reducible to judgments of logical validity 
or** judgments of factual validity of sentences. 



* The notion of 'systemic import' is borrowed from Carl Hempel's 

concept of 'systematic import of scientific concepts' ([6], pp. Sl-S?) , 
and liberally adapted to fit our own perspectives cf system in education. 

Or is used here in the inclusive sense. 
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Judgments Of Meaning 

To insure against expending energies in the conduct of 
empirical tests which are incapable of interpretation, there 
must exist some warranted conditions for differentiating edu- 
cational programs that are capable of interpretable empirical 
test from those which are not. 

The conditions of interpretable empirical te^t are, in 
fact, conditions of adequate meaning • These conditions include 
requirements as to (a) the forms of the claims, (b) the inter- 
nal consistency of the various extra-logical components, and 
(c) the possibility of instantiating each extra-logical com- 
ponent. Judgments made in reference to each of these require- 
ments must be made on logical grounds alone. 

The conditions, each of which is necessary to empirical 
import, are delineated as ''conditions of adequate educational 
program meaning" in the following display. 
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CONDITIONS OF ADEQUATE EDUCATIONAL PROGRAM MEANING 


C[l]: 


Each curricular claim is synonomous with some 
sentence of the form: 

For each pupil X, where X fulfills conditions C; 
If both the teacher (s) and a set of pupils of 
which X is a member , act under rules R, 
then X will (probably) attain goal-state S. 


C[2] : 


Each instructional claim is synonomous with some 
sentence of the form: 

For each pupil X, where X satisfies conditions I, 
If the teacher (s) acts under rules T, in relation 
to a set of pupils of which X is a member, 
then X will (probably)' attain goal-state G. 


C[3]: 


No extra-logical component of the claims 

{ (J uti I U CJ ut UiX I U I uTlo U 1' l^uUTlLL U / Lf ft U LL O ft O Lt 

logical contradiction. 


Cl4]: 


Each statement which contains as a predicate -~ 
an extra-logical component of the claim^ is 
capable^ in principle^ of empirioal test to 
determine whether its implications are 
realized. 


Cl5] : 


No extra-logical component may contain a statement ^ 
of which a pupil is not the subject^ that is 
factually contravalid. 
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Judgments Of Systemic Import 

''Systemic import'' refers to the interconnections among 
the various parts of a structure. A set of unrelated elements 
lacks systemic import. The systemic import of an educational 
program can be assessed from two perspectives: (1) the re- 
lationships among the various claims that constitute the pro- 
gram, and (2) the relationships between these claims and the 
professional framework that has governed the framing of those 
claims . 

The statements representing 'an educational program, classi- 
fied as either curricular or instructional claims, should be 
systemically related. These within-program conne'ctions should 
include at least: (a) between goal-state relationships, (b) 
between rule-set relationships, and (c) relationships between 
qualifying components and goal-state components. 

Formulation and validation of an educational program are 
fundamental responsibilities of professional educators. Under 
this perspective, the form and substance of an educational pro- 
gram, the manner of its realization, and the methods of valida- 
tion must be consistent with the bases of the education profes- 
sion. These professional bases include the aims and functions 
of education (i.e. pragmatic base), credible concepts and prin- 
ciples (i.e. conceptual base), and value orientations and ethical 
norms (i.e. value base) ([12], Part One). Therefore, an 
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educational program should be judged on the basis of relations 
between statements contained in the program and statements 
which represent the professional bases of education. 

The systemic import of an educational program should be 
judged in reference to at least the following conditions. 



CONDITIONS OF ADEQUATE SYSTEMIC IMPORT 
OF AN EDUCATIONAL PROGRAM 



C[l]: Each ourrioular goal-state must be consistent with 
the educational aims ' to which the profession is 
committed, 

C[2]: All curricular and instructional rules must be 

permissible in reference to the value orientations 
and ethical norms of the profession. 

C[3]: The statements representing any (ostensibly) de- 
rived goal^state (curricular or instructional) 
must be: 

(a) synonomous with the statements representing 
the primitive goal-state^ or 

(b) logica lly valid in reference to statements 
representing the primitive goal-state under 
the transformation rules of either class 
logic or propositional logic . 

C[4]: The statements representing any (ostensibly) de- 
rived set of rules (curricular or instructional) 
must be: 

(a) synonomous with the statements representing 
the primitive rules; 

(b) logically valid in reference to statements 
representing the primitive rules under the 
transformation rules of propositional logic 
or^ deontic logic; or 

(c) logically valid in reference to statements 
representing the primitive rules taken in 
conjunction with additional auxiliary 
hypotheses . 



* Or is used here in the inclusive sense. 
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C[S]: For any set of sequentially related curricular 
claims^ the goal-state conditions of any prede- 
cessor claim must be contained within the set of 
qualifying conditions of its successor claim. 

C[6]: ^lo curricular goal-state may contain a sentence 
which is the logical contradiction of a sentence 
contained in another curricular goal-state , 

C[?]: Ilo instructional strategy (i.e. set of instruction- 
al rules) may constitute a logical contradiction of 
the set of curricular rules under which that 
strategy is invoked, 

Cl8]: Each instructional strategy must be logically 
relevant to the set of curricular rules under 
which that strategy is invoked (i.e. the in- 
structional strategy should specify teacher 
actions to be taken in fulfillment of the cur- 
ricu lar rules) . 

C[9]: The probability of pupil attainment of the 

curricular goal-state^ given instructional goal- 
state attainment^ is greater than the probability 
of curricular goal-state attainment^ given non- 
attainment of the instructional goal^state . 
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Judgments Of Potential Factual Validity 

Assuming that we are governed by the conditions of ration- 
al action, then two additional kinds of a priori judgments must 
be made. These are judgments as to whether (a) there is good 
reason to believe that the curricular rules and associated set 
of instructional rules, if invoked, are likely to lead to attain- 
ment of their respective goal-states, and (b) there is good 
reason to believe that the proposed educational program is likely 
to lead to benefits not available under existing educational 
programs . 

The relationships specified by curricular and instructional 
claims, taken separately or collectively, are empirical rela- 
tionships. The propositions offered as reasons in support of 
the belief that these relationships will prove factually valid 
must be grounded — either directly or indirectly — in confirm- 
Infi observational evidence. Consequently, these supporting 
reasons can provide crude in:^rmation as to the likelihood that 
the curricular or instructional claims will prove factually valid 
even before direct empirical tests are conducted. Hence, judg- 
ments as to the quality of these reasons are called judgments 
of ''potential factual validity.** The conditions of adequate 
potential factual validity are set forth in the following 
display. 
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CONDITIONS OF ADEQUATE POTENTIAL FACTUAL VALIDITY 
OF AN EDUCATIONAL PROGRAM 



Cll]: The empirical relation between the set of instruct 
tional rules and an instructional aoal^state . as 
specified by the instructional claim in which 
those extra^logical components are housed^ must 
be supported by sound reasons.'^ 

C [2] : The empirical relation between the set of curric- 
ular rules and a curricular goal^state ^ as specie 
fied by the curricular claim in which those extra^ 
logical components are housed^ must be supported 
by sound reasons. * 

C[Z]: Instructional goal^state attainment^ by pupils 

satisfying the instructional qualifying conditions ^ 
must be potentially effective in relation to attain- 
ment of the associated curricular goal^state . 

C[4]: Lower^level curricular goal^state attainment^ by 

pupils satisfying the lower^level curricular qual^ 
ifying conditions^ must be potentially effective 
in relation to attainment of the higher-level 
curricular goal^state . 



These reasons should consist of empirical propositions 
for which there is some reasonable degree of factual 
confirmation , and may be drawn from education or any 
relevant field of behavioral or natural science, 
praxiology, or technology. 
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A Posteriori Aspects Of Educational Program Validation 



" The Logic Of Testing A Curricular Claim 



Suppose that we have formulated a particular educational 
program (i.e. set of claims) including the precise articulation 
of the various statement sets which constitute their extra- 
logical components. Suppose also that the set of claims satis- 
fies all the foregoing a priori tests. Only at this point is 
it reasonable to conduct empirical trials to determine the 
factual validity of the claims. 



In order to make a justified judgment as to the factual 
validity of a specified curricular claim, denoted by C, a 
number of things must be accomplished. These aspects (borrowed 
from Carl G. Hempel's, Philosophy of Natural Science ([6], 
pp. 6-25)) include the following: 

a. Idenf if ication of significant auxiliary 
hypotheses , A^^ A 2^ Ag^ . . . j i4„. 

b. Specification of the test implication, Tj 
which is to be judged factually valid^ 
factually contravalidj or factually inde^ 
terminate on the basis of observation 
sentences . 

c- Formation of observation sentences on the 
basis of senee-^data ([17] , pp. 7-16). 

d- Judgment of factual validity of the test 
implication, Tj on the basis of its rela- 
tions to the set of observation sentences. 

e. Judgment of factual validity of the (more 
general) curricular claim, Cj taken in 
conjunction with auxiliary hypotheses 

* * * ^ ^n* 
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Even* though the curricular goal-state, curricular rules, 
and curricular qualifying conditions are specified in curricular 
claim (7, the claim still contains variables, A particular pupil 
(i,e, the replacement for X) , a particular set of pupils of 
which X is 3. member, and a particular teacher or set of teachers 
have not been specified in (7, The curricular claim is suffi- 
ciently general to be tested over a wide range of particular 
teacher and pupil sets. 

A test implication, T, or "statement describing the observ- 
able consequences to be expected,*' can be framed by merely 
plugging in particular pupil names and particular teacher names 
for the variables of claim C. It should be noted that this 
example represents the simplest of cases • The curricular claim 
C might contain, in other cases, highly abstract extra-logical 
components, whereas the extra-logical components of T might be 
framed in more elementary terms. 

Formation of a test implication typically demands at least 
tacit assumption of certain premises, or auxiliary hypotheaee y 
in addition to the hypothesis being tested. In reference to 
our present purpose, these auxiliary hypotheses should include 
the relevant set of instructional claims (i.e. hypotheses), 
denoted by J^^ Jgj I j ly^t which are construed as 
components of the educational program. 
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The empirical test of the curricular claim C now involves 
the auxiliary hypotheses (instructional claims) Ijy 1 2^ ••• * 

Ij^'* and the test implication T. The logic of the test can be 
displayed in the following way ([6], pp, 22-25): 

IJ[ C is true, and Ijs 1 2^ ^3^ true, then 

T is true. 

T is not true (i.e. T is judged to be factually contra- 
valid on the basis of its relations 
with— a set of -obser-vatian sentences )~— — 

^Jy ^ -^k true (i.e. they are 

not all factually valid) 

If the test implication T is judged to be factually 
contravalid on the basis of observation sentences which are 
themselves assumed to be factually valid, then the inference is 
made that either C or some subset of the auxiliary hypotheses 
1 23 ^3^ > both, is factually contravalid. As a 

practical procedure, such auxiliary hypotheses should be checked 
during the course of the empirical test period. Grounds for 
rejecting curricular claim C exist only when there is assurance 
that each of the auxiliary hypotheses is valid, and also assur- 
ance that the observation sentences are valid. 



* Analysis of the manner in which instructional claim validity is 

• confounded with curricular claim validity will be the concern of 
future conceptual investigations. 
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Decision Rules For Judging Faotua I Validity 
Of singular Currioular Or Instructional Claims 

Determination of the factual validity of an educational 
program (a system of claims) involves (1) judgments of singu- 
lar curricular or instructional claims, and (2) judgments of 
the set of related claims. This section deals with the first 
aspect, namely, judgments of the factual validity of singular 
curricular or instructional claims, while the following section 
is concerned with judgments of factual validity of an educa- 
tional program. 

Acceptance or rejection of a curricular or instructional 
claim is made on the basis of judgments as to whether conditions 
defined by extra-logical components of the claim have been 
fulfilled. The realization of any extra-logical component of 
a claim may be judged to be either consistent, inconsistent, or 
indeterminate with respect to predictions and explanations de- 
rived from the extra-logical component. The factual validity 
of a curricular or instructional claim may be judged to be 
either factually valid, factually contravalidj or factually 
indeterminate y where that decision depends upon the judgments 
made as to realization of the means and qualifier components of 
that claim. 

The relationships between the judgments of factual validity 
of the claim and judgments as to realization of its extra-logical 
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components are shown in Figure 3. This figure, in effect, sets 
forth the decision rules for determining the validity of a claim 
(curricular or instructional). 

If any of the extra-logical components are judged indeter- 
minate, then the judgment as to the validity of the curricular 
or instructional claitn is indeterminate. The figure does not 
distinguish among the many possible indeterminate combinations, 
since the judgment of the validity of the claim is always the 
same, namely ''indeterminate. 
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• .Decision Rules For Judging Factual Validity 
Of An Educational Program 

Once the judgments of factual validity of each derived or 
partially derived curricular claim have been made, and the valid- 
ity of associated instructional claims has also been determined, 
then judgments of the factual validity of the program-level cur- 
ricular claims can be made. Once judgments of the latter type 
are made, one has the information necessary for making judgments 
as to the factual validity of an educational program. This sec- 
tion deals with this aspect; namely, judgments of the factual 
validity of an educational program (a system of claims). 

The factual validity of an educational program may be 
judged to be either, factually validj factually contravalidj 
or factually indeterminate^ where that decision depends upon 
the judgments made as to the factual validity of the singular 
curricular claims and component sets of instructional claims. 

In judging the validity of educational programs, a distinc- 
tion must be made between two different cases: (1) the case 
where the program contains curricular claims which are derived 
from a set of primitive claims, and (2) the case where the pro- 
gram contains curricular claims which are partially derived 
from a set of primitive claims. This is due to the fact that the 
auxiliary hypotheses must be taken into account in the second case 
(where the program contains a set of partially derived curricular 
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claims) ;* there are no such auxiliary hypotheses in the first 
case . 

The decision rules for judging the factual validity of an 
educational program in reference to a judgment of only one de- 
rived curricular claim are shown in Figure 4. This is the sim- 
plest of possible cases, i.e. the inference requires the fewest 
judgments. This type of educational program, however, is perhaps 
the least likely to exist in a practical situation; ordinarily 
lower-level curricular cliims are more likely to be "partially 
derived" in the sense that they utilize information not contained 
in the program-level curricular claim. 

Figure 5 displays the decision rules for judging the factual 
validity of an educational program in reference to a judgment of 
only one partially derived curricular claim. These decision rules 
include judgments of all components contained in Figure 4, plus 
judgments of the set of auxiliary hypotheses. 

Figures 4 and 5 give the decision rules for judging the 
factual validity of an educational program which contains only 
one derived or partially derived curricular claim, respectively. 
However, an educational program is most likely to contain more 
than one derived or partially derived curricular claim. In this 
case, judging the factual validity of a program requires consid- 
eration of the factual validity of every curricular claim it con- 
tains. Figures 6 and 7 display the decision rules for judging 
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the factual validity of an educational program which contains 
two or more derived curricular claims, and two or more partially 
derived curricular claims, respectively. 

• 

Under these decision rules, a program with an invalid com- 
ponent is an invalid program.* Judging an educational program is 
somewhat akin to judging a chain in that its strength cannot be 
greater than the weakest link. It must be recalled that an edu- 
cational program has been construed as a system of interrelated 
curricular and instructional claims. The view of a program as a 
system demands evaluations which include judgments of that system 
as a whole. 

These decision rules are different from those ordinarily 
applied to testing a system of propositions of descriptive science. 
They differ in the significance attached to a judgment that a 
lower-level hypothesis is factually indeterminate in reference 
to the observational evidence. The judgment that a derived 
hypothesis of a scientific system is ''factually indeterminate" 
does not entail the judgment that the whole system of related 
propositions is also factually indeterminate. The curricular 
claims (and component instructional claims) of an educational 
program, however, are connected in ways that, in some respects, 
are more complex and bind them more tightly than the propositions 
of scientific systems. 

* This assumes no component is indeterminate . 
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The claims of an educational program, for example, are 
connected so that the goal-state of one claim becomes a quali- 
fying condition of another claim; moreover, the goal-states of 
the lower level curricular claims are cumulative in the sense 
that their cumulative acquisition must be the empirical equiva- 
lent of program goal-state attainment. 

From another perspective, the related claims which con- 
stitute an educational program are analogous to the working 
parts of a complex manufacturing system. If any component of 
the manufacturing system is not functioning properly, adequate 
goods cannot be produced. Uncertainty as to the quality of any 
component part produced and assembled at some point of the pro- 
cess yields uncertainty as to the quality of the completed 
product , 

The interrelated sets of decision rules that have been dis- 
played are useful in clarifying both the basic decisions to be 
made, and the grounds for making those decisions. To this point 
we have formulated principles for making judgments under three- 
value logics, where the values are "valid,** contravalid, *' and 
"indeterminate," As is true in the application of any set of 
principles to the empirical world, we will — in practice — be 
confronted with the practical problems of implementing this three- 
value model within real educational circumstances where values 
are continuous. Consequently, there will be some degree of 
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uncertainty associated with every judgment of an empirical 
statement . 

In practice, judgments must be made **on balance;'^ that is 
by weighing all the relevant evidence, particularly where that 
evidence is always incomplete, subject to a variety of kinds of 
sampling and observer errors, and where the evidence may be in- 
consistent. The decisions made are, in the last analysis, value 
judgments that should take into account such considerations as 
the consequences of an error judgment in favor of the relevant 
claim or program, and the consequences of an error judgment 
against the relevant claim or program. 
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SUMMARY AND ASSESSMENTS 

• « 

SummaTij 

A comprehensive conceptual framework for the validation 
of educational programs has been proposed. The conceptual 
framework includes the concept of an educational program as 
a system of curricular and instructional claims and methods 
of evaluation which entail a wide range of judgments of both 
logical validity and factual validity. Judgments of logical 
validity are made in reference to the internal structure of 
each of the claims and its various components, the relation- 
ships among the claims constituting a program, and the rela- 
tionships between the component claims of a program and the 
bases of the profession under which the program is formulated 
and operates. 

The logic of reaching judgments of factual validity of 
the individual curricular and instructional claims, and of the 
program as a whole, has been described. Judgments of factual 
validity of the various component claims, and of the program, 
are made on the basis of observations of pupil performance and 
teacher action to determine whether those behaviors correspond 
to that which is entailed by fulfillment of the goal, rule, and 
qualifier components of the claims. 
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Assessments of the Proposed Conoepts and Prinaiples 

In assessing the merit of the proposed conceptual frame- 
work and method for educational program evaluation, one must 
answer the quest ion : Does the method ino lude all the kinds of 
evaluative judgments construed as desirable under the existing 
credible evaluation models? The proposed program evaluation 
methods include a comprehensive range of evaluative judgments 
classified in other models as '^process and product'* or ''forma- 
tive and summative" evaluations; such classes of evaluation 
as "input" and "context" are included as well. 

A second question that should be asked is: Does the pro- 
posed evaluative framework and method lead to warranted evalua- 
tive judgments that are of a significantly different type and 
quality from the evaluative judgments made under the existing 
credible models? The proposed ways of program evaluation in- 
clude important types of evaluation which no existing model 
for educational evaluation takes into account (although they 
are commonplace in the sciences). For example, the proposed 
ways of evalution lead to examination and judgment of the 
relations among the various goal-states and procedural rule- 
sets contained within a program; and to examination and judg- 
ment of the internal structure of those goal-states and pro- 
cedural rules as well. Further, the concept of a program as 
a system of propositions — each having a goal component, a 
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rule component and a qualifier component — leads to evaluative 
judgments as to whether the teacher acts in a manner consistent 
with the procedural rules (i.e. curricular rules or instruction- 
al rules); and evaluative judgments as to whether the pupil acts 
in a manner that is consistent with the curricular rules. These 
judgments are necessary to warranted .judgments of the effective- 
ness of a program or any of its curricular or instructional com- 
ponents. 

The third question that should be asked is: Is there good 
reason to believe that the proposed eva luative framework and 
method will lead to the development of more effective educational 
programs? In answering this question, one must first note the 
lack of clarity as to the meaning of ''educational program" under 
other evaluation models; the prevailing concepts of "educational 
program" may be useful in ordinary talk about programs, but they 
are inadequate for the purpose of credible program evaluation. 
It should be noted that fundamental inadequacies exist in the 
current evaluation models; recall, for example, the defect identi- 
fied by Stufflebeam as cited on page 1 of this paper. The ef- 
fectiveness of an educational program is determined to a great 
extent by the evaluative principles and standards that have 
guided its formation and cosequent reconstructions. 

Where program formation and modification is governed by 
the evaluative schema proposed in this paper, the disconnected- 
ness and contradictions that ordinarily exist will be eliminated. 
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Since the program is construed as a system of claims undef 
which both teachers and pupils act, it becomes possible to 
determine whether a defect lies in the design (represented by 
the component curricular and instructional claims) or in the 
manner in which the design is implemented by teachers and 
pupils. Under other constructs of ''educational program,'' 
statement and action are confounded, and it is impossible to 
detect the points of inadequacy. From the perspective of 
empirical test , application of the principles set forth in 
the proposed evaluative schema leads to warranted judgments 
as to program effectiveness. These principles are grounded 
in the credible principles of hypothesis testing of the 
empirical sciences; the inferences entailed by prevailing 
models of educational evaluation do not satisfy these condi- 
tions . 

A Concluding Note 

The constructions of this paper will enable the serious 
educational program evaluator or developer to become clear 
as to the object under evaluation or construction; and to 
reach justified evaluative judgments in reference to both log- 
ical relationships and empirical effectiveness. Fundamental 
defects can be identified. The authors claim that the funda- 
mental defects of educational program evaluation can be resolved 
only by pursuing a path of the sort described in this paper. 
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